Device and method for navigating and/or guiding the path of a vehicle, and vehicle

ABSTRACT

The invention relates to a device (80) and to a method for navigating and/or guiding the path of a vehicle (200), in particular of a wheelchair, and to a vehicle, in order to achieve increased safety and comfort during operation of the vehicle. The device (80) comprises at least one first sensor, in particular an inertial navigation unit (11), which is designed and arranged so as to detect at least one first body part, in particular an absolute position and/or position and/or rotation and/or translation, of a passenger of the vehicle (200) and to output first sensor signals (S1), and a second sensor, in particular an image-based sensor (21, 22), which is designed and arranged so as to detect at least the first body part of the passenger and/or its features and to output second sensor signals (S2). A control unit (100) receives the sensor signals (S1, S2), ascertains first control signals (ST1) based on the first sensor signals (S1), and it is ascertained, based on the second sensor signals (S2), whether the first control signals (ST1) comply with at least first reliability criteria. A safety mode is adopted when the first reliability criteria are not complied with.

The invention relates to a device and a method for navigating and/orguiding the path of a vehicle, in particular a wheelchair, and to avehicle.

A special wheelchair control system for an electric wheelchair is knownfrom EP 3 646 127. This comprises an input element, for example dataglasses attached to the head of a passenger, and an adapter box fortransmitting the data from the input element to an input/output moduleof the electric wheelchair. To generate travel commands, head movementsof a passenger are detected by motion-detecting sensors when the dataglasses are used and assigned to desired travel directions and travelspeeds. Control signals generated on the basis of the travel commandsare transmitted to the input/output module of the wheelchair in order tocontrol the wheelchair.

EP 3 076 906 relates to a control unit for a personal vehicle. Thecontrol unit is able to provide a control signal for the vehicle withthe aid of two independent motion sensors by evaluating the determinedrelative orientations of the independent motion sensors to each other.As a result, a desired driving command from the passenger can bedetected even in the presence of disturbance variables, such asinclines, declines, or unpaved roads. If the evaluation of theorientation sensors identifies already known gestures, these are used asvehicle control signal. Additional, external sensors can be used toconfigure and determine the position and attitude of the vehicle.

The described prior art is problematic in many respects. It is true thatin many cases motion-based or orientation-based control can be performedusing data from at least a first motion sensor. Also, some embodimentsallow detection of motion patterns to initiate, for example, anemergency stop, emergency call, or launch of external applications.However, a failure of the first motion sensor may already lead to afailure of the vehicle control system. For example, in a vehicle controlsystem based purely on head orientation, even a cramp or short-termstiffening of the neck can lead to a loss of control by the passenger.Control commands not desired by the passenger, for example triggered byerrors in the first motion sensor, can also be passed on to theactuators of the vehicle and set it in motion or leave it in motion.

Based on this prior art, it is the object of the present invention toprovide a device and a method which creates increased safety as well asincreased comfort during the operation of a vehicle. In particular,increased fail-safety is to be made possible, the transmission ofundesired control commands to the vehicle is to be avoided and thecomfort during use of the vehicle is to be improved. The object issolved by the device according to claim 1, the vehicle according toclaim 15 and the method according to claim 16.

The object is solved in particular by a device for navigating and/orguiding the path of a vehicle, in particular a wheelchair, wherein thedevice comprises at least:

-   -   a first sensor, in particular an inertial navigation sensor,        which is designed and arranged to detect at least one first body        part, in particular a position and/or rotation and/or        translation, of a passenger of the vehicle and to output first        sensor signals;    -   a second sensor, in particular an image-based sensor, which is        designed and arranged to detect at least the first body part of        the passenger and/or their features, in particular their        absolute position and/or position and/or rotation and/or        translation, and to output second sensor signals;    -   a control unit which is designed to receive the first and second        sensor signals, determine first control signals for controlling        the vehicle based at least on the first sensor signals,        determine whether the first control signals meet at least first        reliability criteria based at least on the second sensor        signals, and adopt a safety mode if the control unit determines        that the first control signals do not meet at least the first        reliability criteria.

One idea of the invention arises from the fact that every sensor issubject to a rate of error or failure that cannot be completelyeliminated. In this regard, it is particularly advantageous inapplications where there are significant, negative effects of sensorfailure to provide suitable measures for dealing with sensor failures.The present invention addresses this circumstance by not directly usingfirst sensor signals from at least a first sensor to determine a controlsignal for the vehicle, but instead using at least second sensorsignals, in particular image-based sensor signals, from a second sensorto determine at least the first reliability criteria, such that a safetymode is adopted if at least first control signals determined using thefirst sensor signal do not meet the first reliability criteria.

A safety mode can be understood as a restricted operating range of thedevice, wherein the scope of restriction is determined on the one handbased on sensor-detectable, physical states of a passenger. On the otherhand, reduced accuracy and/or reliability of the sensors and actuatorsused by the device can also lead to a restriction of the operating rangeof the device, e.g. driving at reduced speed.

The reliability criteria are intended to define thresholds above whichthe safety mode, i.e. the restricted operating range of the device, mustbe adopted. With respect to the detectable, physical states of thepassenger, the thresholds may be general thresholds or individualizedthresholds. Individualized thresholds may be necessary if the externallyperceptible expressions of physical states, facial expressions orgestures, of a passenger are significantly restricted. This may be thecase, for example, in the event of (partial) paralysis of a body part,such as the face. In this case, it is possible that thresholds arelearned by the device by repeating certain facial expressions orgestures several times. It is also possible for thresholds to be adaptedduring operation of the device, for example if it is known that apassenger usually begins to tremble during prolonged operation of thevehicle as a result of general exertion.

The reliability criteria can also define thresholds for the requiredaccuracy and/or reliability of individual sensors. In addition,permissible tolerances in the deviations of the same physicalparameters, determined by different sensors, can be defined. An exampleof this is the permissible position deviation of the iris in aneye-feature-based control of the vehicle, in which two different sensorsdetect the passenger's eye.

Using at least the second sensor signals, conclusions can be made as towhether the passenger is capable of controlling the vehicle. How todecide whether a passenger is capable of controlling the vehicle dependson the chosen embodiment of the present invention. It is possible todetermine whether the passenger is able to control the vehicle bycomparing the second sensor signals with predefined first reliabilitycriteria.

For example, in the case of a passenger, at least the second sensor candetect a strong tilt of the head to one side over a longer period oftime as a result of progressively decreasing neck muscle strength. Withhead-orientation-based control applied, this could result in apotentially dangerous circular motion ride. In such a case, the safetymode can adjust the permissible operating range of the device so thatthe vehicle brakes to a standstill by outputting a second controlsignal.

Braking to a standstill is a possible measure influencing drivingdynamics in response to physical states that can be detected by sensors.

It is also possible to determine a safe stopping position before thevehicle stops using advanced sensors for position determination or anenvironment-sensing sensor assembly and to control this stoppingposition. Here, too, the device generates second control signals thatare not the same as the first control signals.

The first or second control signals may be control signals thatinfluence driving dynamics and are transmitted to vehicle actuators.According to the invention, it is also possible that the control signalsprovide data for device or passenger monitoring or choices for vehiclemodes to be adopted to a wearable computer system (wearable) or apassenger's cell phone. The control signals can also be designed forforwarding to a remotely located computer system (backend) for remotedevice or passenger monitoring.

In safety mode, the device allows the operating range to be adjusted sothat an emergency mode is initiated. This allows predeterminedindividuals to be notified of the initiated safety mode. If required,the sensed physical conditions of the passenger can also be transmitted,for example to perform a human diagnosis of the passenger's physicalconditions from a distance.

Also, a safety mode may be initiated due to insufficient sensorcapability. It is conceivable that one or more of the sensors mayprovide their own accuracy or reliability analysis data to the controlunit, whereupon the one or more sensors may not be used to determine thefirst control signals or reliability criteria.

When first control signals or second control signals classified as validare generated by the device, they are transmitted to vehicle actuators,in particular controllable motors, wherein the vehicle actuators arearranged to drive the vehicle and are designed to receive and processthe control signals from the device.

According to the invention, a first body part may be any body part ofthe human body capable of performing movements.

In one embodiment, the first body part is the passenger's head. In thisembodiment, the device comprises as a first sensor an inertialnavigation unit, arranged on or in the wearable computer system(wearable), which is designed to detect at least the head, in particularan absolute position and/or position and/or rotation and/or translation,and to output first sensor signals.

The second sensor detecting the passenger's head may be an image-basedsensor, such as an image-based front sensor or an image-based rearsensor of a cell phone. However, it may also be an additionalimage-based sensor attached to the vehicle that detects the passenger'shead. The control unit may be configured to receive the first and secondsensor signals and determine first control signals for controlling thevehicle based on the first sensor signals.

Based on the second sensor signals, it can be determined whether thefirst control signals meet at least first reliability criteria. For thispurpose, the first reliability criteria are determined based on thesecond sensor signals. The safety mode may be adopted if the controlunit determines that the first control signals do not meet at least thefirst reliability criteria.

Basic data can be determined on the basis of the second sensor signals.These basic data can be geometric dimensions and/or size ratios and/orfeatures of the face and/or gestures, wherein the first reliabilitycriteria are determined using the basic data.

These geometrically or mathematically expressed properties of the basicdata can characterize facial expressions perceivable by humans and allowconclusions to be drawn about the emotional state perceived by thepassenger during operation of the vehicle. On the one hand, if there isa high probability of a state of happiness, a state of relaxation, or asimilar state associated with positive mood on the part of thepassenger, the first reliability criteria are said to be met. On theother hand, if there is a high probability of the passenger beinganxious, in a state of surprise, in a state of fear, or in a similarstate associated with a negative mood, the first reliability criteriashall mean a failure to meet the first reliability criteria.

According to the invention, the image-based analysis can also detectredness, enlargement or reduction of the eyes, including the pupils, aswell as the opening or closing of the eyes or the mouth and lips,including inferable characteristics such as shortness of breath,increased pulse or increasing, dangerous, physical exertion of theperson. Detection of such physical conditions also means failure to meetthe first reliability criteria.

In addition, first reliability criteria may not be met if the firstcontrol signals detect control commands from the passenger that conflictwith at least the second sensor signals, i.e. one sensor determines adesired travel to the left, while at least one other sensor determines adesired travel to the right.

If the first reliability criteria are not met, the safety mode isinitiated. According to the invention, it is consequently possible notto transmit an already generated control signal to vehicle actuatorsresponsible for vehicle control due to the violation of firstreliability criteria. This arrangement can ensure that obviouslyrecognizable, critical physical conditions of the passenger aredetected, the vehicle is stopped and/or an emergency call modedownstream of the safety mode enables rapid assistance for the passengerby contacting an emergency contact. In addition, third parties and thepassenger cannot be further physically harmed as a result.

In one embodiment, the control unit is (further) configured to determinethe first control signals in addition to the first sensor signals of theinertial navigation unit attached to or in the wearable using also thesecond sensor signals of the image-based sensor. In this case, thesafety mode is adopted if the first or the second reliability criteriaare not met. In this embodiment, the second reliability criteria arebased on accuracy and/or reliability analysis of at least the first andthe second sensor signals.

Accuracy and/or reliability analysis can be performed using sensorself-diagnostic data provided with the sensor signals.

The accuracy and/or reliability analysis may further be determined usingthe first and second sensor signals, respectively, of one or more of thefeatures of absolute position and/or position and/or rotation and/ortranslation of the passenger's head, and based on their matching, thereliability of the two sensors may be estimated.

If the matching of the feature or features used to control the vehiclematches within a predefined tolerance or predefined tolerances, thefeature values of the inertial navigation unit are used to determine thefirst control signal. If one or more of the tolerances are exceeded,self-diagnostic data from the image-based sensor is used, and if thereliability is poor, for example due to poor image quality, a signalabout the poor reliability is communicated to the passenger via thewearable or to the passenger's cell phone. Generating control signalsbased on the first sensor signals is further enabled. If the image-basedsensor is sufficiently reliable, a safety mode is again initiated,because the control unit assumes a failure of the inertial navigationunit. In this case, the safety mode initiates an immediate stop.Continuous analysis of the sensors enables safe operation of the vehicleand immediate action is taken to minimize the consequences of damage inthe event of sensor faults.

In a further embodiment, the fulfillment of the first reliabilitycriteria is determined by implementing machine learning, in particularby classification, preferably by a support vector machine (SVM), or byimplementing a neural network, in particular a convolutional neuralnetwork (CNN). The embodiment has the advantage of the individualclassifiability of the image-based face recognition to the specificpassenger. Furthermore, the aforementioned artificial intelligence meansgenerally achieve better hit rates in the field of image-based facerecognition and are more reliable than conventional classificationmethods, for example decision trees.

For these artificial intelligence approaches, it is necessary to providepre-labeled image data, in particular image data with associated personfacial expressions, for classification of the expressed mood of thepassenger. This labeled image data can be an image data set ofarbitrarily selected persons, an individual image data set of thepassenger, or a mixed image data set.

In a further embodiment, a third sensor, in particular a vitality datasensor, is arranged on a body part or between two body parts to detectvitality parameters, in particular heart rates, of the passenger and tooutput third sensor signals. The control unit receives the third sensorsignals, wherein if a vitality parameter value range is not met by thethird sensor signals, the second reliability criteria are not met. As aresult, the safety mode is adopted.

Since vitality parameters can be recorded on any part of the human body,the body part to which the vitality data sensor is attached can be anypart of the body. In particular, an arrangement on the neck, in the ear,around the wrist, around the chest or around the abdomen is preferable.

Also, the vitality data sensor may be a two-part sensor, such as a cheststrap and a wrist transmission unit.

Vitality data, moreover, are not limited to heart rates. Continuousmonitoring of blood pressure, endogenous blood oxygen content, bloodcount, blood glucose, saliva or other endogenous parameters that can bemeasured directly or indirectly by the vitality data sensor can also bemade possible.

In a further embodiment, instead of the inertial navigation unit, thefirst sensor is an eye feature detection unit. This is attached to or inthe wearable and is designed to detect at least one eye of the passengerand/or its features, in particular the absolute position and/or positionand/or rotation and/or translation. The eye feature detection unit mayoutput first sensor signals, wherein the control unit receives the firstsensor signals and determines the first control signals for controllingthe vehicle using the first sensor signals. The eye feature detectionunit may be an RGB image-based eye feature detection unit or an infraredlight-based eye feature detection unit.

By analyzing the accuracy and/or reliability of the first sensor signalsof the eye feature detection unit and the second sensor signals of theimage-based sensor, it is possible to check whether the secondreliability criteria are met. In particular, the control unit mustdetermine whether deviations between the first and second sensor signalsdetecting the eye features are detectable.

A deviation of one or more of the features relevant for the generationof first control signals absolute position and/or position and/orrotation and/or translation of at least one eye of the passenger must bewithin the permissible deviations or tolerances of the secondreliability criteria.

If one or more of the tolerances are exceeded, self-diagnostic data fromthe image-based sensor is relied upon, and if the reliability is poor,for example due to poor image quality, a poor reliability signal istransmitted to the passenger on the wearable or on the passenger's cellphone. Generating first control signals based on the first sensorsignals is further enabled.

If tolerances are not met despite sufficient reliability of theimage-based sensor, a safety mode is initiated because the control unitassumes errors in the eye feature detection unit. The safety mode isinitiated and at least causes the vehicle to stop.

Continuous analysis of the sensors enables safe operation of the vehicleand immediate action is taken to minimize the consequences of damage inthe event of sensor faults. Safety is further enhanced by the thirdsensor, the vitality data sensor. This is because unless the thirdsensor signals meet a vitality parameter value range, the secondreliability criteria are not met.

In one embodiment, the first sensor is a speech input sensor attached toa head-hold mechanism, a wearable, or the vehicle to provide speechinput to the passenger. The speech input sensor is configured to outputfirst sensor signals, wherein the control unit receives the first sensorsignals and determines the first control signals for controlling thevehicle using the first sensor signals.

The control unit is designed to use an implementation of neuralnetworks, preferably recurrent neural networks (RNN), for example anLSTM or a GRU, or convolutional neural networks (CNN) when processingthe first sensor signals to determine the first control signals. Inparticular, for passengers with limited speech capabilities,self-learning neural networks allow speech recognition to be adapted tothe passenger's voice. On the one hand, the control commands given byspeech input can result in first control signals for (starting) orstopping the vehicle. On the other hand, voice inputs can be used tooperate a menu navigation, which can be displayed to the passenger bythe wearable by projecting a head-up display. This allows furthercommands to be issued, for example to open doors, order elevators, pressbuttons, operate switches or other external applications.

The control unit is further designed to determine the first controlsignals in addition to the first sensor signals of the speech inputsensor using also second sensor signals of a second sensor, preferablyan image-based sensor. In this case, the second sensor is directed atthe passenger's head and is designed to detect features of the absoluteposition and/or position and/or rotation and/or translation of one orboth eyes and to output the second sensor signals.

As a third sensor, the vitality data sensor is arranged on a body partor between two body parts to detect vitality parameters, in particularheart rates, of the passenger and output third sensor signals. Thecontrol unit receives and processes the third sensor signals, and if avitality parameter value range is not met by the third sensor signals,the second reliability criteria are not met. The safety mode is adopted.

An eye feature detection unit, attached to or in the wearable, is usedas the fourth sensor. Like the second sensor, this is designed to detectat least one eye of the passenger and/or its features, in particular itsabsolute position and/or position and/or rotation and/or translation,and to output fourth sensor signals, wherein the control unit receivesthe fourth control signals and uses them to determine the first controlsignals for controlling the vehicle. The eye feature detection unit maybe an RGB image-based eye feature detection unit or an infraredlight-based eye feature detection unit. Consequently, in thisembodiment, the first, second, and fourth sensor signals are used todetermine the first control signals.

This results in a redundant system, with the use of an infraredlight-based eye feature detection unit resulting in diversitaryredundancy in eye feature detection by the second and fourth sensors.The safety mode is adopted in this embodiment when the first or secondreliability criteria are not met.

The control unit can determine basic data using the second sensorsignals. This basic data can be geometric dimensions and/or size ratiosand/or features of the face and/or gestures, wherein the firstreliability criteria are determined using the basic data.

These properties of the basic data, expressed geometrically ormathematically, characterize the facial expressions that can beperceived by humans and allow conclusions to be drawn about theemotional state perceived by the passenger during the operation of thevehicle. On the one hand, if there is a high probability of a state ofhappiness, a state of relaxation, or a similar state associated withpositive mood on the part of the passenger, the first reliabilitycriteria are said to be met. On the other hand, if there is a highprobability of the passenger being anxious, in a state of surprise, in astate of fear, or in a similar state associated with a negative mood,the first reliability criteria shall mean a failure to meet the firstreliability criteria.

It is also possible that when positive sentiment is detected, feedbackoccurs within the control unit and the control unit starts a recordingmode. In this case, at least the passenger's head movements detected bythe second, image-based sensor are recorded and used to train a machinelearning algorithm in order to recognize the passenger's learnedgestures at a later time and to be able to interpret them as controlcommands. For example, one such gesture might be a quick shake of thehead to trigger the vehicle to stop. This form of gesture recognitionmakes the use of the vehicle more personalized for the passenger.

The fulfillment of the first reliability criteria is checked byimplementing machine learning, in particular by classification,preferably by a support vector machine (SVM), or by implementing aneural network, in particular a convolutional neural network (CNN). Atleast when using CNNs, probabilities for the recognized facialexpressions are also evaluated by the control unit to determine whetherthe first reliability criteria are met.

Both machine learning methods and neural network implementations arebased on image-based, labeled training data, which includes image dataof faces or facial features. Also, this training data may include imagesof the passenger's face. This allows the adaptability of the facialexpression detection algorithm to the specific passenger.

According to the invention, the image-based analysis can also detectredness, enlargement or reduction of the eyes, including the pupils, aswell as the opening or closing of the eyes or mouth and lips, includinginferable characteristics such as shortness of breath, increased pulseor increasing, dangerous exertion of the person. Detection of suchphysical conditions also means failure to meet the first reliabilitycriteria.

In this embodiment, the second reliability criteria are based onaccuracy and/or reliability analysis of the first, second, and fourthsensor signals. In particular, an accuracy comparison of the detectedeye or iris features of the second, image-based sensor with those of thefourth sensor, the eye feature detection unit, is performed.

Based on the accuracy and/or reliability analysis of the second sensorsignals of the image-based sensor and the fourth sensor signals of theeye feature detection unit, the control unit can check whether thedeviation of one or more of the features absolute position and/orposition and/or rotation and/or translation of the passenger's eye iswithin one or more tolerances.

Should one or more of the tolerances be exceeded, self-diagnostic datafrom the image-based sensor is relied upon, and in the event of poorreliability, for example due to poor image quality, a signal of poorreliability is transmitted to the passenger via the wearable or to theircell phone. The generation of control commands based on the fourthsensor signals is further enabled.

Unless a low reliability of the second, image-based, sensor can bedetermined when one or more of the tolerances is exceeded, the safetymode is adopted and the vehicle is at least stopped. The reason for thisis that if the reliability of the second sensor is sufficient, thecontrol unit assumes that there is an error in the eye feature detectionunit.

If the first and second reliability criteria are not violated,eye-feature-based control of the vehicle can be started by assuming aninitial state, such as fixing a point in the center of the pair of eyes.Also, a sequence of gestures, for example, blinking the eyes three timesor certain predefined eye movement patterns, may signify the start ofeye-feature-based control. Also, the first speech input sensor can beused to detect a start command.

Control is then performed by assigning a gaze direction to the desireddirection of movement. The vehicle can be stopped by detecting apredefined gesture, such as closing the eyes, or blinking the eyes, fromthe second sensor or the fourth sensor. Also, the vehicle can be broughtto a stop by voice input using the first sensor, the voice input sensor.

Continuous analysis of the sensors enables safe operation of the vehicleand immediate action is taken to minimize the consequences of damage inthe event of sensor faults. Safety is also enhanced by the use of thethird sensor, the vitality data sensor, because unless the third sensorsignals meet a vitality parameter value range, the second reliabilitycriteria are not met. The safety mode is adopted.

In one embodiment, a fifth sensor, a brain-control unit interface orinput device, is arranged on and/or in the passenger's head instead ofthe fourth sensor, the eye feature detection unit. This is designed tooutput fifth sensor signals, with the control unit receiving the fifthsensor signals and using them to determine the first control signals forcontrolling the vehicle. The brain-control unit interface is used as theprimary data source for generating the first control signals. Further,the image-based sensor is used to determine at least the firstreliability criteria, and if the first reliability criteria are not met,the safety mode is initiated.

Predefined control commands are associated with the fifth controlsignals in the control unit, such that a passenger thought detected bythe fifth sensor signals is associated with a desired control command.Furthermore, abstract thought patterns of the passenger detected in thefifth sensor signals can be provided to a machine learning algorithm inthe control unit, wherein the learning algorithm learns from theprovided data and determines the first control signals based on detectedcontrol commands of the passenger.

Furthermore, however, the signals can also be used to determine whetherthe passenger has panicked, for example, because he or she is in adangerous physical condition.

By combining with the first speech input sensor, the second image-basedsensor, and the third sensor of this embodiment, the vitality datasensor, dangerous situations can be classified more easily and drivingthe vehicle becomes safer.

In one embodiment, all previously used sensors are combined forproviding an even safer and more reliable solution. As a first sensor,the inertial navigation unit, mounted on or in the wearable, is used todetect the absolute position and/or position and/or rotation and/ortranslation of the passenger's head and to transmit first sensor signalsto the control unit and, using them, to determine the first controlsignals for controlling the vehicle. The second, image-based, sensor candetect the passenger's head and transmit second sensor signals to thecontrol unit, using which the first reliability criteria are determinedand the first control signals for controlling the vehicle aredetermined.

A third sensor, the vitality data sensor, transmits third sensor signalsto the control unit. These describe vitality parameters, in particularheart rates, of the passenger, wherein the second reliability criteriaare not met if the third sensor signals do not comply with a vitalityparameter value range.

As a fourth sensor, an eye feature detection unit is used, attached toor in the wearable, and designed to detect one or both eyes of thepassenger, in particular their absolute position and/or position and/orrotation and/or translation of the passenger, and to output fourthsensor signals, wherein the control unit receives the fourth sensorsignals and, using them, determines the first control signals forcontrolling the vehicle.

Based on the accuracy and/or reliability analysis of the first sensorsignals of the inertial navigation unit and the second sensor signals ofthe image-based sensor, the control unit can check whether the deviationof one or more of the features absolute position and/or position and/orrotation and/or translation of the passenger's head is within one ormore tolerances.

Based on the accuracy and/or reliability analysis of the second sensorsignals of the image-based sensor and the fourth sensor signals of theeye feature detection unit, the control unit can check whether thedeviation of one or more of the features absolute position and/orposition and/or rotation and/or translation of the eye or eyes of thepassenger is within one or more tolerances.

The fifth sensor, in particular the brain-control unit interface orinput device, is arranged on and/or in the passenger's head and isdesigned to output fifth sensor signals, wherein the control unitreceives the fifth sensor signals and determines the first controlsignals for controlling the vehicle using them. The fifth sensor signalsmay include control commands from the passenger to start and stop.

The sixth sensor used is the voice input sensor, which can in particularbe a head-hold mechanism voice input sensor, a wearable voice inputsensor, or a vehicle voice input sensor and can be designed to outputsixth sensor signals, wherein the control unit receives the sixth sensorsignals and uses them to determine the first control signals forcontrolling the vehicle. Through the voice input sensor, passengercommands for starting or stopping the vehicle can be transmitted to thecontrol unit by the sixth sensor signals.

If one or more of the tolerances are not met, the second reliabilitycriteria are not met. If the first or second reliability criteria arenot met, the device adopts safety mode.

If the reliability criteria are met, first control signals for movingthe vehicle can be transmitted from the control unit to the vehicleactuator. The fourth sensor signals define the direction of travel aspart of the first control signals, while the first sensor signals definethe desired speed in the direction of travel.

In a further embodiment, a first environment-sensing sensor assembly, inparticular an ultrasonic sensor assembly and/or a LIDAR sensor assemblyand/or an image-based sensor assembly and/or a RADAR sensor assembly, isalso attached to the vehicle for sensing the environment and isconfigured to output first sensor assembly signals, wherein the controlunit receives first sensor assembly signals and determines the firstcontrol signals using them.

The environment can also be sensed by an image-based rear sensor and/ora wearable image sensor, and corresponding sensor signals can betransmitted to the control unit. Using these, the control unit candetermine the first control signals.

In this case, the recorded environmental information can be evaluated bythe control unit and the safety mode can be adopted in the event of acritical approach to obstacles, slopes, inclines or similar.

In one embodiment, the wearable and/or a cell phone is configured andarranged to output seventh sensor signals, in particular passengerdestination input sensor signals, wherein the control unit haspreviously transmitted available destination input selection data to thewearable and/or the cell phone for passenger destination input.

The destination input selection data is generated by evaluation of thesensor signals of at least the first environment-sensing sensor assemblyby the control unit. The targets can basically be all elements whosecontour can be detected by a RADAR and/or LIDAR and/or ultrasonic and/orimage-based sensor assembly and which are in the passenger's field ofview.

For image-based sensor data, trained convolutional neural networks(CNNs) can be used. The image-based training dataset required to trainone or more CNNs includes a variety of different labeled image datacaptured indoors as well as outdoors by different cameras and underdifferent lighting situations, so that the CNNs can recognize a varietyof objects sensed by the environment-sensing sensor assembly and providethem to the passenger as destination input selection data.

Destination input selection data is projected into the passenger's fieldof view when the wearable is used, modeled on a head-up display. Cellphones allow a screen display. The control unit receives the seventhsensor signals and determines the first control signals using them.

The control unit has the task of calculating a trajectory to thedestination, providing suitable first control signals and, during thejourney to the destination, making adjustments to the trajectory basedon the real-time data from the environment-sensing sensors, for exampledue to the changing environment, and providing adjusted first controlsignals.

When using LIDAR or RADAR sensor assemblies, three-dimensional,environment-mapping maps are generated. The three-dimensional pointclouds of these maps can be assigned image features of captured imagesof the environment by machine learning. This makes it possible togenerate a three-dimensional trajectory to the target object in theenvironment. The vehicle moves along this trajectory by providing thegenerated first control signals to the vehicle actuator to minimize thedistance to the target location or object. When the minimum distance tothe target is reached, the vehicle stops.

It is also possible to project destination input selection data to thepassenger in real time on his or her head-up field of view in thewearable by arranging the wearable image sensor and/or at least one,preferably image-based, environment-sensing sensor assembly to alignwith the environment based on the direction of the passenger's directionof view and to transmit first sensor assembly signals to the controlunit. Using the first sensor assembly signals, the control unitdetermines destination input selection data and transmits it to thewearable for projection into the head-up display. Preferably, this isdone over an LTE or a 5G connection.

The passenger can thus perform a target selection directly based on thereal-time images or a real-time video, and seventh sensor signals, inparticular passenger destination input sensor signals, are transmittedfrom the wearable to the control unit. The control unit can determinethe first control signals using the seventh sensor signals.

The second, third and fifth sensor signals are used by the control unitto provide feedback on whether the vehicle is on the right path to itsdestination from the passenger's point of view. The passenger's facialexpression, heart rate and thoughts are monitored, with safety modebeing adopted if a critical physical condition of the passenger isdetected based on the sensor signals. The automated control of aselected target object can also be interrupted or terminated at any timeby manual override by the passenger.

In one embodiment, an eighth sensor, in particular a position sensor,for example a GPS sensor or a sensor for locating inside buildings, isarranged and designed to output eighth sensor signals, with the controlunit receiving the eighth sensor signals and determining the firstcontrol signals using them.

The position sensor allows the display of a map showing the currentposition of the vehicle or passenger using the head-up projection of thewearable or on their cell phone. The use of a map in conjunction withthe displayed destination input selection data allows the selection ofdestinations or objects that can no longer be sensed by environmentalsensing sensors. A target can be selected by guiding a virtual mousepointer via eye control in the head-up projection of the wearable or byselection on the cell phone.

Based on the passenger's destination input, the seventh passengerdestination sensor signals are provided to the control unit. Using themap and the eighth position sensor signals, a trajectory to the targetlocation or object can be determined and corresponding first controlsignals can be provided.

Thus, the control unit has the task of calculating a trajectory to thedestination, providing suitable first control signals, and, during thejourney to the destination, making adjustments to the trajectory basedon the real-time data from the environment-sensing sensors, for exampledue to the changing environment, and providing adjusted first controlsignals. After reaching the target location or target object, thevehicle stops.

The second, third and fifth sensor signals can be used by the controlunit to provide feedback on whether the vehicle is on the right path toits destination from the passenger's point of view. The passenger'sfacial expression, heart rate and thoughts are monitored, with safetymode being adopted if a critical physical condition of the passenger isdetected on the basis of the sensor signals.

Also, the automated control of a selected target object can beinterrupted or terminated at any time by manual override by thepassenger.

In one embodiment, the device comprises a remote communication device,for example an LTE or 5G communication device or a mobile telephonycommunication device, arranged on the vehicle and designed to performsignal exchange with the control unit and external mobile telephonyprovider devices. In this case, the wearable is also connected to thecontrol unit via an LTE or 5G connection so that (video) communicationcan be performed or images or videos captured by the wearable imagesensor can be provided to others in personal messages or via socialnetworks.

Furthermore, in case of an initiated safety mode, for example due to adetected critical physical condition of the passenger, at least oneemergency contact can be contacted automatically.

In one embodiment, the device also comprises vehicle dynamics sensors,in particular wheel speed sensors and/or inertial navigation units withintegrated accelerometers and/or rate-of-rotation meters and/or compass,arranged on the vehicle and designed to output ninth sensor signals,wherein the control unit receives the ninth sensor signals and uses themto determine the first control signals. In particular, this sensorsystem allows the vehicle dynamics to be controlled by feeding backactual vehicle dynamics sensor variables to the control unit in order toenable more precise implementation of the desired control commandscompared with a control system.

In another embodiment, the image-based front sensor for detecting thepassenger's face is replaced by an image-based vehicle sensor, whereinthe image-based vehicle sensor transmits image-based sensor signals tothe control unit and, using the image-based sensor signals, the controlunit determines the first control signals and evaluates whether thefirst reliability criteria are met.

Further advantageous embodiments result from the subclaims.

In the following, the invention is also described with regard to furtherfeatures and advantages on the basis of exemplary embodiments, which areexplained in more detail with reference to the figures, wherein:

FIG. 1 shows a flowchart of the device according to an exemplaryembodiment;

FIG. 2 shows a vehicle with wearable, inertial navigation unit,image-based sensor, control unit and vehicle actuators;

FIG. 3 shows the vehicle according to FIG. 2 with vitality data sensor;

FIG. 4 shows the vehicle shown in FIG. 3 without inertial navigationunit, with eye feature detection unit;

FIG. 5 shows the vehicle shown in FIG. 4 , with wearable voice inputsensor;

FIG. 6 shows the vehicle according to FIG. 3 , without wearable, withoutinertial navigation unit, with brain control unit interface or inputdevice, without wearable voice input sensor, with vehicle voice inputsensor;

FIG. 7 shows the vehicle according to FIG. 6 , with wearable, withinertial navigation unit, with eye feature detection unit, with speechinput sensor;

FIG. 8 shows the vehicle according to FIG. 7 , with wearable imagesensor, with environment-sensing sensor assembly; and

FIG. 9 shows the vehicle according to FIG. 8 with position sensor, withremote communication device, with wearable voice input sensor, withoutvehicle voice input sensor.

In the following description and in the drawings, the same referencesigns are used for identical and similarly acting parts.

FIG. 1 shows a flowchart of a device 80 for navigating a wheelchair 200(see FIG. 2 ).

In one exemplary embodiment, the device 80 comprises two sensors. Thefirst sensor is an inertial navigation unit 11 attached to a wearablecomputer system, wearable 10, on the head of a wheelchair user of awheelchair 200 (e.g., FIG. 2 ). For example, it is part of Google Glassworn by the wheelchair user. The inertial navigation unit 11 detectsposition and rotation of the head of the wheelchair user and outputsthis data as first sensor signals S1. An image-based front sensor 21 ofa cell phone 20 (FIG. 2 ) serves as a second sensor, wherein the cellphone 20 is fixed to a component of the wheelchair 200 such that thefront sensor 21 detects the head of the wheelchair user. Image data ofthe front sensor 21 generated in this process is output by the cellphone 20 as second sensor signals S2.

A control unit 100 is communicatively connected, e.g. via Bluetooth, toGoogle Glass and receives the first sensor signals S1. The second sensorsignals S2 can be transmitted to the control unit 100, in particular toa sensor signal receiving unit 101, via a USB connection. In a controlsignal determination unit 102 of the control unit 100, based on thefirst sensor signals S1, first control signals ST1 for controlling thewheelchair 200 are generated. Based on the second sensor signals S2, areliability criteria checking unit 103 determines whether the firstcontrol signals ST1 satisfy first reliability criteria.

A safety mode is adopted when the reliability criteria checking unit 103determines that the first control signals ST1 do not comply with thefirst reliability criteria.

The reliability criteria checking unit 103 determines basic data basedon the second sensor signals S2. The basic data includes geometricdimensions, size ratios, and features of the face. These geometricallyor mathematically expressed features of the basic data characterize theperceptible facial expressions of a person, thereby allowingclassification of the emotional state or emotions felt by the wheelchairuser during operation of the wheelchair 200.

Devices for classifying the emotions of a person on the basis ofrecognized geometric dimensions, proportions and features of the faceare known. For example, IN 00554CH2014 A describes a method and a devicewith which facial expressions of persons that have changed compared topreviously determined, neutral facial expressions can be recognized andassigned to one or more emotions. The teachings of IN 00554CH2014 A useso-called constrained local model (CLM) methods for recognizing faces inorder to then determine dimensions, proportions and shapes of featuressuch as eyes, nose, mouth or chin. Furthermore, a support vector machine(SVM) can be trained to infer actions such as closing an eye or openinga mouth from the previously recognized geometric facial features using avariety of previously labeled image data of faces. Finally, astatistical procedure (Discriminative Power Concept) is used todetermine the probability of a particular emotion of a person byevaluating how likely an action is if a particular emotion is presentminus the probability of that action if that emotion is not present.

Specifically, this may mean that when upwardly drawn corners of themouth are detected relative to a known, neutral mouth position, a stateof happiness of the wheelchair user is determined, since none of theknown, further emotions are characterized by upwardly drawn corners ofthe mouth. The IN 00554CH2014 A identifies the emotions anger, fear,happiness state, surprise state, disgust, sadness as recognizable. Inaddition, probabilities of the individual emotions are output in eachcase.

The exemplary embodiment offers the possibility and thus the advantageof the individual classifiability of the image-based facial featurerecognition to the concrete wheelchair user, provided that the supportvector machine SVM is (also) trained with labeled image data of facialmuscle movements of the wheelchair user. Moreover, the captured featuresof the face underlying the classification of the emotion can beevaluated in real time. For this purpose, these are superimposed on theimage data sequence (video sequence) of the second sensor signals S2 ofthe image-based front sensor 21.

Alternatively, according to another exemplary embodiment as shown inFIG. 1 , the basis of the classification of the wheelchair user'semotions is an implementation of a convolutional neural network (CNN)applied to the second sensor signals S2 in the reliability criteriachecking unit 103. For this artificial intelligence approach, it isnecessary to provide pre-labeled image data, i.e., image data withassociated emotions of a person, for classification of the expressedemotion of the wheelchair user. In this case, the geometric features ofthe face, their dimensions and size ratios are not visible during thetraining due to the abstract representation of the trained weights ofthe neural network.

The hit rate and thus the quality of the emotion recognition is to beassessed on the basis of a test data set. The convolutional neuralnetwork is designed in such a way that a probability value is assignedto each of the number of emotions of the wheelchair user to berecognized. The use of the neural network has the advantage that similarto the human perception of emotions in faces, not one or a few featuresof the face are taken into account, but the emotion of the wheelchairuser is determined in the interaction of all image data transmitted bythe second sensor signals S2. Thus, the CNN training data can alsoimplicitly consider person-specific features of the face, such as smilelines, for emotion recognition without explicitly characterizing them inthe training data.

Regardless of the choice of emotion classification means, SVM or CNN, ifthe wheelchair user is highly likely to be in a state of happiness, astate of relaxation, or a similar state associated with positiveemotion, the first reliability criteria should be met.

However, a detected positive sentiment does not only mean that the firstreliability criteria are met. In addition, a time series recording ofthe position and rotation data of the head of the wheelchair user isalso performed in the control signal determination unit 102, which ismade available to a machine learning algorithm. This algorithm candetect and process the individual movement sequences of the head of thewheelchair user mapped by the data, whereby the wheelchair control isadapted to the personal requirements.

It is possible to determine for each wheelchair user the maximumoccurring inclinations of the head in the directions front/rear orleft/right during operation of the wheelchair 200 as well as thecorresponding rotation speeds. Thus, for a wheelchair user with stillavailable but severely restricted freedom of movement of the neck, evena slight inclination of the head in one direction can mean a maximummovement of the wheelchair in the desired direction. The additionalrecording of the rotation speeds via the position angle of the head alsomakes it possible to avoid interpreting head inclinations thatinevitably result from illness-related trembling or nervous wheelchairusers as control commands. This also allows wheelchair users withunintentional or uncontrolled jerky head movements to have a smoother,more individual driving behavior of the wheelchair.

Regression algorithms lend themselves as machine learning algorithms forthis task. As a result, a characteristic of the control of thewheelchair 200 desired by the wheelchair user need not be based on onlya single recorded position-time or rotational speed-position function.Rather, multiple, temporally spaced recorded functions can be used tomap an averaged, individual characteristic of the control of thewheelchair 200.

If the first reliability criteria are met, control is effected bytilting the head upward to cause the wheelchair 200 to move forward orbackward, depending on the presetting. A leftward or rightward tilt ofthe head will cause the wheelchair to move leftward or rightward,respectively. To stop the wheelchair, the head is moved to a previouslydefined normal or neutral position.

The first reliability criteria are intended to signify a failure to meetthe first reliability criteria when there is a high probability of aninability to control wheelchair 200 such as apprehension, a state ofsurprise, a state of anxiety, a state of dissatisfaction, or a similarstate of the wheelchair user associated with negative emotion. Forexample, downturned corners of the mouth signal a dissatisfaction state.States of surprise and anxiety can be identified by the detection ofwide open eyes or mouth. An inability to control the wheelchair 200 isinferred from closing the eyes for a period longer than a blink period.The training data provided to the convolutional neural network (CNN)takes into account this mapping of facial expressions to emotions orstates. The same applies to the exemplary embodiment for theclassification of emotions by the support vector machine (SVM).

The safety mode is a restricted operation range of the device 80,wherein in the safety mode, second control signals ST2 are output fromthe reliability criteria checking unit 103. In this case, if there is aninability to control the wheelchair 200, e.g. if the wheelchair user hasclosed his eyes, the second control signals ST2 will cause thewheelchair to slow down and stop.

When the probability of a state of surprise, a state of fear, or a stateof apprehension of the wheelchair user is increased, the reliabilitycriteria checking unit 103 uses the first sensor signals S1 of theinertial navigation unit 11 to determine the second control signals ST2.If the determined position or rotation of the wheelchair user's head iswithin an inadmissible range of values, the second control signals ST2cause the wheelchair 200 to stop. On the other hand, if the position androtation of the wheelchair user's head is within an admissible range ofvalues, only the maximum speed of the wheelchair 200 is reduced becausethe reliability criteria checking unit 103 assumes that it has beenunreasonably high for the wheelchair user so far.

The second control signals ST2 are not equal to the first controlsignals ST1, but both can be output from the device 80 by the controlsignal output unit 104. Only the control signals that are valid at onetime are output.

This arrangement ensures that recognizably negative emotions of thewheelchair user or an inability to control the wheelchair 200 aredetected and the safety mode is adopted. In addition, third parties andthe wheelchair user cannot be further physically harmed as a result.

When the first reliability criteria are met, the first control signalsST1 are determined by the control signal determination unit 102 of thewheelchair 200 using the first sensor signals S1. Depending on thedefault setting, tilting the head of the wheelchair user upward meansmoving forward or moving backward, while tilting the head to the sidemeans orienting the wheelchair 200 to that side.

In another exemplary embodiment of the wheelchair 200 in FIG. 2 , thecontrol unit 100 is further configured to determine the first controlsignals ST1 in addition to the first sensor signals S1 from the inertialnavigation unit 11 attached to the wearable 10 using also the secondsensor signals from the image-based front sensor 21. In this regard, thesafety mode is adopted when the first reliability criteria or the secondreliability criteria are not met. The safety mode is also adopted whenboth reliability criteria are not met. In this exemplary embodiment, thesecond reliability criteria are based on accuracy and reliabilityanalysis of the first sensor signals S1 and the second sensor signals S2by the reliability criteria checking unit 103.

The accuracy and reliability analysis is performed using the firstsensor signals S1 and the second sensor signals S2, which includefeature values of the position and rotation of the head of thewheelchair user. Based on their matching, the reliability of theinertial navigation unit 11 and the image-based front sensor 21 isdetermined. If the position and rotation features used for controllingthe wheelchair 200 match within predefined tolerances, the featurevalues of the inertial navigation unit 11 are used for determining thefirst control signals ST1. The control of the wheelchair 200 isperformed as shown in the preceding exemplary embodiment.

If one or more of the tolerances are exceeded, the second sensor signalsS2 of the image-based front sensor 21 are used again and, in the eventof poor accuracy and reliability, for example due to poor image quality,a signal about the poor reliability is transmitted to the wheelchairuser on his cell phone 20 and displayed there. The poor image qualitycan be detected by the fact that the algorithm used to detect thewheelchair user's emotion assigns only low probabilities to each of theindividual emotions, e.g., happiness state, dissatisfaction. Thegeneration of first control signals ST1 based on the first sensorsignals S1 is further enabled.

If the reliability of the front sensor 21 is sufficient, the safety modeis initiated because the reliability criteria checking unit 103 assumesa failure of the inertial navigation unit 11. The wheelchair 200 isstopped.

By continuously matching the feature values of the inertial navigationunit 11 and the image-based front sensor 21, safe operation of thewheelchair 200 is enabled and immediate action is taken to minimize theconsequences of damage in the event of sensor failure.

FIG. 3 shows another exemplary embodiment in which, in addition, avitality data sensor 40 is arranged as a third sensor on the wrist ofthe wheelchair user in order to detect the heart rate of the wheelchairuser and output third sensor signals. The control unit 100 receives thethird sensor signals transmitted, for example, via a Bluetoothconnection by means of the sensor signal receiving unit 101, and thereliability criteria checking unit 103 further checks whether there is ahigh probability of a state of concern or anxiety of the wheelchair userwhen the third sensor signals do not comply with an allowable heart ratevalue range. If this is also the case, the second reliability criteriaare evaluated as not met. As a result, the safety mode is adopted. Thesecond control signals ST2 cause the wheelchair 200 to stop.

FIG. 4 shows another exemplary embodiment similar to the precedingexample. However, instead of the inertial navigation unit 11, this hasan eye feature detection unit 12 as the first sensor. This is attachedto the wearable 10 and connected to it, for example via cabling, and isdesigned to detect the position and translation of the iris of an eye ofthe wheelchair user and to output first sensor signals S1, wherein thesensor signal receiving unit 101 receives the first sensor signals S1after they have been forwarded by the wearable 10 and uses them todetermine the first control signals ST1 for controlling the wheelchair200.

Based on the accuracy and reliability analysis of the first sensorsignals S1 of the eye feature detection unit 12 and the second sensorsignals S2 of the image-based front sensor 21, the reliability criteriachecking unit 103 checks whether the second reliability criteria aremet. The reliability criteria checking unit 103 determines whether thereare deviations of the feature values position and translation of theiris of an eye.

Deviations of these characteristic values must be within predefined,permissible tolerances of the second reliability criteria.

If one or more tolerances are exceeded, the sensor signals of theimage-based front sensor 21 are used, as in the preceding exemplaryembodiment, and if the accuracy and reliability are poor, for exampledue to poor image quality, a signal about the poor reliability isprojected to the wheelchair user through the wearable 10 into his fieldof view. In addition, the information is transmitted by the controlsignal output unit 104, for example via a Bluetooth connection, to hiscell phone 20. Generating first control signals ST1 based on the firstsensor signals S1 is further enabled.

If the tolerance is not met, i.e., if the allowed position deviation ofthe iris position detected by both sensors is not met despite sufficientreliability of the image-based front sensor 21, the safety mode isinitiated because the reliability criteria checking unit 103 assumeserrors in the eye feature detection unit 12. The safety mode causes thewheelchair 200 to stop.

When the iris positions and translations of an eye determined by thesecond and fourth sensors match within tolerances, a final positionvalue is determined by the control signal determination unit 102 bycalibrating the eye movement range. In this process, the maximummovement of the eyes to the left, right, up, and down is determined,thereby enabling eye position proportional control of the wheelchair200. If a reliable iris position cannot be determined in this process,the second reliability criteria are not met, the safety mode is adopted,and the wheelchair 200 is stopped.

If the first and second reliability criteria are not violated, the eyefeature-based control of the wheelchair 200 is started by assuming aninitial state, for example, by fixing a point in the center of the pairof eyes. Also, a sequence of gestures, for example, eye blinks or apredefined eye movement pattern, may signify the starting of the eyefeature-based control by the control signal detection unit 102. In thisregard, an eye movement pattern is a sequence of gaze directions definedand executed by the wheelchair user.

The wheelchair 200 is controlled by assigning a viewing direction to thedesired direction of movement. The control of the direction of movementof the wheelchair 200 is done by a gaze to the left signifying arotation of the wheelchair 200 to the left and a gaze to the rightsignifying a rotation of the wheelchair to the right. The wheelchair 200is stopped by detecting a predefined gesture, such as closing the eyes,or blinking the eyes several times, by the first sensor, the eye featuredetection unit 12, or by the second sensor, the image-based front sensor21.

Continuous analysis of the sensors enables safe operation of thewheelchair 200 and immediate action is taken to minimize theconsequences of damage in the event of sensor failure. Safety is furtherenhanced by the vitality data sensor 40. This is because unless thethird sensor signals meet a vitality parameter value range, the secondreliability criteria are not met. The safety mode is adopted.

FIG. 5 shows a further, modified exemplary embodiment in which the firstsensor is a wearable voice input sensor 61 that is attached to thewearable 10 and connected thereto, for example, by cabling. The wearablevoice input sensor 61 is configured to output first sensor signals S1,wherein the sensor signal receiving unit 101 receives the first sensorsignals S1 after being forwarded by the wearable 10 and determines thefirst control signals ST1 for controlling the vehicle 200 in the controlsignal determination unit 102 by using them.

The control signal determination unit 102 is configured to use animplementation of a recurrent neural network (RNN) when processing thefirst sensor signals S1 to determine the first control signals ST1.These algorithms for analyzing completed, sequential speech sensorsignals allow for training that is more efficient in terms of time, forexample, compared to fully-connected neural networks (FCN), as well asimproved recognition of the speech inputs of the wheelchair user withcomparable computational effort.

Especially for wheelchair users with limited speech capabilities, theself-learning neural networks allow the speech recognition to be adaptedto the voice of the wheelchair user. On the one hand, the controlcommands issued by speech input can result in the first control signalsST1 for driving (off) or stopping the wheelchair 200. On the other hand,voice inputs are used to operate a menu navigation, which is displayedto the wheelchair user by the wearable 10 by projection of an ordinaryhead-up display. This allows commands to be issued, for example, to opendoors, order elevators, press buttons, operate switches.

The control unit 100 is further configured to determine the firstcontrol signals ST1 in addition to the first sensor signals S1 of thewearable voice input sensor 61 using also second sensor signals S2 ofthe image-based front sensor 21. In this case, the second sensor isdirected at the head of the wheelchair user and is designed to detectthe position and translation of both eyes or their irises and to outputthe second sensor signals S2.

As a third sensor, the vitality data sensor 40 is arranged on the wristof the wheelchair user for determining the heart rate, wherein thissensor as well as the control unit 100 are designed to provide thefunctionalities according to the preceding exemplary embodiments.

The eye feature detection unit 12, attached to the wearable 10, is usedas the fourth sensor. Like the second sensor, the image-based frontsensor 21, the eye feature detection unit 12 is configured to detect theposition or translation of an eye or iris of the wheelchair user andoutput fourth sensor signals, wherein the sensor signal receiving unit101 receives the fourth control signals and, using them, the controlsignal determination unit 102 determines the first control signals ST1for controlling the vehicle 200. The eye feature detection unit 12 is aninfrared light-based eye feature detection unit.

Consequently, in this exemplary embodiment, the first sensor signals S1,the second sensor signals S2, and the fourth sensor signals are used todetermine the first control signals ST1. This results in a redundantsystem, and the use of the infrared-light-based eye feature detectionunit 12 results in diversitary redundancy in eye feature detection bythe second and fourth sensors.

The safety mode is adopted in this exemplary embodiment when thereliability criteria checking unit 103 determines that the firstreliability criteria or the second reliability criteria are notsatisfied. Satisfaction of the first reliability criteria is determinedin the reliability criteria checking unit 103 using at least the basicdata of the second sensor, the image-based front sensor 21, as shown inthe preceding exemplary embodiments.

As in previous exemplary embodiments, the evaluation of the secondreliability criteria comprises a reliability and accuracy analysis.Based on the second sensor signals S2 of the image-based front sensor 21and the fourth sensor signals of the eye feature detection unit 12, thereliability criteria checking unit 103 checks whether the deviation ofthe position or translation of an eye of the wheelchair user is withinthe respective permissible tolerance.

If one or more of the tolerances are exceeded, as described in thepreceding exemplary embodiments, the sensor signals of the image-basedfront sensor 21 are used, and if the accuracy and reliability are poor,for example due to poor image quality, a signal about the poorreliability is transmitted to the wheelchair user via the wearable 10 orto his cell phone 20. Generating first control signals ST1 using thefourth sensor signals by the control signal determination unit 102 isfurther enabled, and the first control signals ST1 are output by thecontrol signal output unit 104.

Unless low reliability of the image-based front sensor 21 is detectedwhen one or more of the tolerances is exceeded, the safety mode isadopted and the wheelchair 200 is stopped. The reason for this is thatif the reliability of the image-based front sensor 21 is sufficient, thereliability criteria checking unit 103 assumes that there is an error inthe eye feature detection unit 12.

The wheelchair is controlled as in the preceding exemplary embodiment.In addition, however, the first sensor, the wearable voice input sensor61, can be used to detect a start command.

Also, the wheelchair 200 is brought to a stop by voice input using thefirst sensor, the wearable voice input sensor 61.

Continuous analysis of the sensors enables safe operation of thewheelchair 200 and immediate action is taken to minimize theconsequences of damage in the event of sensor failure. Safety is furtherenhanced by the use of the third sensor, the vitality data sensor 40,because unless the third sensor signals satisfy a vitality parametervalue range, the reliability criteria checking unit 103 determines thatthe second reliability criteria are not satisfied. The safety mode isadopted according to the conditions of the preceding exemplaryembodiments.

FIG. 6 shows a further exemplary embodiment of the wheelchair 200according to the invention, wherein instead of the fourth sensor, theeye feature detection unit 12, a fifth sensor, a brain control unitinterface or input device 50 is arranged on the head of the wheelchairuser. This is designed to output fifth sensor signals, wherein thesensor signal receiving unit 101 receives the fifth sensor signals, forexample via a Bluetooth connection, and determines the first controlsignals ST1 for controlling the wheelchair 200 in the control signaldetermination unit 102 using them.

A brain control unit interface is known from US 2017/0042439A1. Here, anelectrode strip is arranged around a person's head in such a way thatbrain waves can be measured and subsequently processed. In the process,a mental state or emotion of the person can be determined.

The brain control unit interface or input device 50 is used as theprimary control data source for generating the first control signalsST1. Furthermore, the image-based front sensor 21 is used to determinethe first reliability criteria, and if the first reliability criteriaare not met, the safety mode is adopted. In the case of an additionalelevated heart rate detected by the vitality data sensor 40, thereliability criteria checking unit 103 assumes a critical physicalcondition of the wheelchair user and, furthermore, the secondreliability criteria are not met. The wheelchair 200 is stopped.

In the control signal determination unit 102, the fifth sensor signalsare detected so that a thought of the wheelchair user detected by thefifth sensor is assigned to a desired first control signal ST1.

For this purpose, the brain-control unit interface or input device 50knows predefined control commands that are output as fifth sensorsignals when the wheelchair user thinks of them.

However, the fifth sensor signals include not only predefined controlcommands.

As part of the fifth sensor signals, the abstract thought patterns ofthe wheelchair user detected by the fifth sensor are provided to amachine learning algorithm in the control signal determination unit 102.The learning algorithm learns from the data so that when a previouslyknown thought pattern that can be associated with a control command ofthe wheelchair user is detected, the first control signals ST1 aredetermined based on the detected control command.

The machine learning algorithm is trained by recording, in accordancewith the preceding exemplary embodiments, the emotion of the wheelchairuser in response to a determined first control signal ST1 and alsoprovided to the control signal determination unit 102. The heart rate ofthe wheelchair user is also provided. This allows the machine learningalgorithm to learn which control commands determined based on thethought patterns are judged appropriate by the wheelchair user. Thisallows for customization of the thought-based control of the wheelchair200 for the particular wheelchair user.

However, determining the emotions of the wheelchair user is not limitedin application to training the machine learning algorithm.

Further, the fifth sensor signals are also used to determine whether thewheelchair user has panicked, for example, because the wheelchair useris in a dangerous physical condition. By combining with the firstsensor, the vehicle voice input sensor 60, the second sensor, theimage-based front sensor 21, and the third sensor of this exemplaryembodiment, the vitality data sensor 40, dangerous situations can beclassified more easily and driving the wheelchair 200 becomes safer.

FIG. 7 shows another exemplary embodiment of the wheelchair 200,combining all the sensors used so far to provide an even safer and morereliable solution. The first sensor used is the inertial navigation unit11, attached to the wearable 10, to detect the position and rotation ofthe head of the wheelchair user and transmit first sensor signals S1 tothe sensor signal receiving unit 101 and, using them, determine thefirst control signals ST1 in the control signal determination unit 102for controlling the wheelchair 200. The second sensor, the image-basedfront sensor 21, detects the head of the wheelchair user or its featuresand transmits second sensor signals S2 to the control unit receivingunit 101, and using them, the reliability criteria checking unit 103checks whether the first reliability criteria are satisfied, as knownfrom previous exemplary embodiments. In addition, the first controlsignals ST1 for controlling the wheelchair 200 are determined using thesecond sensor signals S2.

The third sensor, vitality data sensor 40, is arranged and configured asin preceding exemplary embodiments. The fourth sensor, an eye featuredetection unit 12, is attached to the wearable 10, and is configured todetect position and translation of an eye or an iris of the wheelchairuser as known from previous embodiments. The fourth sensor signals areprovided to the control unit 100, so that the first control signals ST1can be determined using them.

Based on the accuracy and reliability analysis of the first sensorsignals S1 of the inertial navigation unit 11 and the second sensorsignals S2 of the second sensor of the image-based front sensor 21, thereliability criteria checking unit 103, as known from previous exemplaryembodiments, checks whether the deviations of the position or therotation of the head of the wheelchair user are within the respectivepermissible tolerances.

In the accuracy and reliability analysis of the second sensor signals S2of the second sensor, the image-based front sensor 21 and the fourthsensor signals of the fourth sensor, the eye feature detection unit 12,the reliability criteria checking unit 103, as known from previousexemplary embodiments, checks whether the deviation of the featuresposition or translation of one eye of the wheelchair user is within therespective tolerances.

The fifth sensor, the brain control unit interface or input device 50,is arranged and configured as known from the preceding exemplaryembodiments. The control signal determination unit 102 determines, usingthe transmitted fifth sensor signals, control commands from thewheelchair user to start and stop the wheelchair 200.

The vehicle voice input sensor 60 is used as the sixth sensor, which isconfigured to output sixth sensor signals, wherein the control unit 100receives the sixth sensor signals and determines the first controlsignals ST1 for controlling the wheelchair 200 using them. Through thevehicle voice input sensor 60, commands for starting or stopping thewheelchair 200 can be transmitted to the sensor signal receiving unit101 by the sixth sensor signals.

Unless one or more of the tolerances are met, the second reliabilitycriteria are not met. If the first or second reliability criteria arenot met, the device 80 adopts the safety mode. If they are met, firstcontrol signals ST1 for moving the wheelchair 200 may be transmittedfrom the control signal output unit 104 to the vehicle actuators 70.

Using the fourth sensor signals, the desired direction of travel isdetermined by the control signal determination unit 102, and using thefirst sensor signals, the desired speed in the direction of travel isdetermined. From this, the first control signals ST1 are determined andoutput to the vehicle actuators 70 by the control signal output unit104.

A forward look of the wheelchair user leads to a forward movement, alook to one side leads to a movement of the wheelchair 200 in therespective direction. Depending on the presetting, a downward pitchingmovement leads to an increase/decrease in the wheelchair speed, while anupward pitching movement has the opposite effect on the wheelchair speedin each case.

This exemplary embodiment also has the advantage over the previous onesof increasing the safety of the wheelchair user, since the user mustlook in the direction of travel.

FIG. 8 illustrates another exemplary embodiment that further comprises afirst environment-sensing sensor assembly 90, a LIDAR sensor assembly.This is attached to the wheelchair 200 for sensing the environment andis designed to output first sensor assembly signals, wherein the controlsignal sensing unit 101 receives first sensor assembly signals via ahigh-speed data line, such as an Ethernet connection, and determines thefirst control signals ST1 using the same.

Furthermore, the perception of the environment is performed by animage-based rear sensor 22 of the cell phone 20 and a wearable imagesensor 13, so that using them the first sensor signals ST1 aredetermined by the control signal determination unit 102. The wearableimage sensor 13 is connected to the wearable 10 via data lines, forexample, so that the wearable 10 transmits its sensor signals to thesensor signal receiving unit 102.

In this case, the detected environment information is evaluated by thecontrol signal determination unit 102 and the reliability criteriachecking unit 103, and the safety mode is adopted when a criticalapproach to obstacles, slopes, inclines or the like is detected. Thiscauses the wheelchair 200 to stop.

In another exemplary embodiment, the wearable 10 and the cell phone 20are configured to output seventh sensor signals, in particular passengerdestination input sensor signals, wherein the control unit 100 haspreviously transmitted available destination input selection data to thewearable 10 and the cell phone 20 for passenger destination input.

The destination input selection data is created by evaluating the sensorsignals of the first environment-sensing sensor assembly 90 by thecontrol unit 100. The destinations can basically be any elements whosecontours can be detected by a LIDAR sensor assembly 90 in combinationwith the further environment-sensing sensors 13, 22 and which arelocated in the environment of the wheelchair user.

Destination input selection data is projected into the wheelchair user'sfield of view, modeled on a head-up display from the wearable 10. Inaddition, the information is displayed on the screen of the cell phone20. The sensor signal receiving unit 101 receives the seventh sensorsignals and determines the first control signals ST1 using them.

In this context, the control signal determination unit 102 has the taskof calculating a trajectory to the destination, providing suitable firstcontrol signals ST1, and, during travel to the destination, makingadjustments to the trajectory, for example as a result of the changingenvironment, based on the real-time data from the environment-sensingsensors 13, 22 and the LIDAR sensor assembly 90, and providing adjustedfirst control signals ST1.

When the LIDAR sensor assembly 90 is used, three-dimensional,environment-mapping maps are generated. Image features of capturedimages of the environment are assigned to the three-dimensional pointclouds of these maps by machine learning. This makes it possible togenerate a three-dimensional trajectory to the target object in theenvironment. The wheelchair 200 moves along this trajectory by providingthe generated, first control signals ST1 to the vehicle actuators 70 tominimize the distance to the destination or target object. When theminimum distance to the target is reached, the wheelchair 200 stops.

Real-time destination input selection data is projected to thewheelchair user in his or her head-up display in the wearable 10 by thewearable image sensor 13 transmitting its sensor signals to the controlunit 100, and the environment-sensing LIDAR sensor assembly 90 isarranged to align with the environment based on the direction of thewheelchair user's direction of view and to transmit first sensorassembly signals to the control unit 100. The control unit determinesdestination input selection data using aforementioned signals andprovides the same to the wearable 10 for projection into the head-updisplay.

As a result, the wheelchair user can perform a target object selectiondirectly based on the real-time images or a real-time video, and seventhsensor signals, in particular passenger destination input sensorsignals, are transmitted from the wearable 10 to the sensor signalreceiving unit 101. The control signal determination unit 102 determinesthe first control signals ST1 using the seventh sensor signals.

The second, third, and fifth sensor signals are used by the controlsignal determination unit 102 to provide feedback on whether thewheelchair 200 is on the correct path to the destination from thewheelchair user's perspective. In this regard, the facial expression ofthe wheelchair user, vitality parameters, and the thoughts of thewheelchair user are monitored, with the safety mode being adopted if acritical physical condition of the wheelchair user is detected based onthe sensor signals. Also, at any time, the automated actuation of aselected target object can be interrupted or terminated by manualoverride by the wheelchair user, for example based on the sensor signalsfrom the inertial navigation unit 11, the eye feature detection unit 12,or the image-based front sensor 21.

FIG. 9 shows another exemplary embodiment according to the invention. Inthis case, an eighth sensor, a GPS position sensor 106, is arranged onthe wheelchair 200 and is designed to output eighth sensor signals, withthe sensor signal receiving unit 101 receiving the eighth sensor signalsand determining the first control signals ST1 using them.

The position sensor 106 allows a map showing the current position of thewheelchair 200 or wheelchair user to be displayed using the head-upprojection of the wearable 10 and on his cell phone 20. The use of a mapin conjunction with the displayed destination input selection dataallows selection of destinations or objects that cannot be sensed byenvironment-sensing sensors 13, 22 or the environment-sensing sensorassembly 90. A target is selected by guiding a virtual mouse pointer viaeye control in the head-up projection of the wearable 10 or by selectionon the cell phone 20.

Based on the destination input from the wheelchair user, the seventhpassenger destination input sensor signals are provided to the controlsignal determination unit 102. Using the map and the eighth sensorsignals of the GPS signal-based position sensor 106, a trajectory to thetarget location or object is determined and corresponding first controlsignals ST1 are provided.

Thus, the control signal determination unit 102 has the task ofcalculating a trajectory to the destination, providing suitable firstcontrol signals ST1, as well as making adjustments to the trajectory,for example due to the changing environment, during the journey to thedestination based on the real-time data from the environment-sensingsensors 13,22 and the environment-sensing sensor assembly 90, andproviding adjusted first control signals ST1. After reaching thedestination or target object, the wheelchair 200 stops.

As described in the previous exemplary embodiment, the second, third,and fifth sensor signals are used by the control signal determinationunit 102 to provide feedback on whether the wheelchair 200 is on thecorrect path to the destination from the perspective of the wheelchairuser. In case of violation of the first or second reliability criteriaknown from previous exemplary embodiments, the safety mode is adopted.

At any time, the automated control of a selected target object can beinterrupted or terminated by manual override by the wheelchair user.

FIG. 9 shows another exemplary embodiment in which a remotecommunication device 107 for transmitting LTE or 5G signals is alsoarranged on the wheelchair 200. This is in signal exchange with thecontrol unit 100 and external mobile phone provider devices. In thiscase, the wearable 10 is also connected to the control unit 100 via anLTE or 5G connection, so that (video) communication can be carried outor images or videos captured by the wearable image sensor 13 can beprovided to others in personal messages or via social networks.

Furthermore, in the event of an initiated safety mode, for example dueto a detected critical physical condition of the wheelchair user, atleast one emergency contact can be contacted automatically. On the onehand, an automated text message is sent. In addition, a video call isstarted.

FIG. 9 also shows another exemplary embodiment of the wheelchair 200, inwhich another inertial navigation unit with integrated accelerometersand rate-of-rotation meters and compass, arranged on the wheelchair 200and designed to output ninth sensor signals, is used as the vehicledynamics sensor 71. The sensor signal receiving unit 101 receives theninth sensor signals and determines the first control signals ST1 usingthem. In particular, this sensor system allows the control of thedriving dynamics by feeding back the driving dynamics sensor variablesto the control signal determination unit 102 to enable a more preciseimplementation of the desired control commands compared to a purecontrol system.

In a further exemplary embodiment, which can be taken from FIG. 9 , theimage-based front sensor 22 for detecting the face of the wheelchairuser is replaced by an image-based vehicle sensor 14, wherein the lattertransmits image-based sensor signals to the control signal receivingunit 101 via a connection, e.g. USB connection, and the control signaldetermination unit 102 uses the image-based sensor signals to determinethe first control signals ST1, and the reliability criteria checkingunit 103 evaluates whether the first reliability criteria have been met.

At this point it should be pointed out that all parts described above,in particular the individual embodiments and exemplary embodiments, areto be regarded in each case individually—even without featuresadditionally described in the respective context, even if these have notbeen explicitly identified individually as optional features in therespective context, e.g. by using: in particular, preferably, forexample, e.g., optionally, round brackets, etc.—and in combination orany sub-combination as independent designs or further developments ofthe invention, as defined in particular in the introduction to thedescription as well as in the claims. Deviations therefrom are possible.Specifically, it should be noted that the word “in particular” or roundbrackets do not indicate any features that are mandatory in therespective context.

LIST OF REFERENCE SIGNS

-   -   10 Computer system that can be worn on the body (wearable)    -   11 An inertial navigation unit    -   12 Eye feature detection unit    -   13 Wearable image sensor    -   14 Image-based vehicle sensor    -   20 Cell phone    -   21 Image-based front sensor    -   22 Image-based rear sensor    -   40 Wearable vitality data sensor    -   41 Ear vitality data sensor    -   50 Brain control unit interface or input device    -   60 Vehicle voice input sensor    -   61 Wearable voice input sensor    -   70 Vehicle actuators    -   71 Vehicle dynamics sensors    -   80 Device for navigation and/or guiding the path and/or        stabilizing a vehicle    -   90 Environment-sensing sensor assembly    -   100 Control unit    -   101 Sensor signal receiving unit    -   102 Control signal determination unit    -   103 Reliability criteria checking unit    -   104 Control signal output unit    -   106 Position sensor    -   107 Remote communication device    -   200 Vehicle, in particular wheelchair    -   ST1 First control signals    -   ST2 Second control signals    -   S1 First sensor signals    -   S2 Second sensor signals

1. A device for navigating and/or guiding the path and/or stabilizing avehicle, the device comprising: at least one first sensor including aninertial navigation unit, which is designed and arranged to detect atleast one first body part, including an absolute position and/orposition and/or rotation and/or translation, of a passenger of thevehicle and to output first sensor signals; at least one second sensorincluding an image-based sensor, which is designed and arranged todetect at least the first body part of the passenger and/or theirfeatures, including their absolute position and/or position and/orrotation and/or translation, and to output second sensor signals; and acontrol unit designed to, a) receive the first and second sensorsignals; b) determine first control signals for controlling the vehiclebased at least on the first sensor signals; c) determine whether thefirst control signals meet at least first reliability criteria based atleast on the second sensor signals; and d) adopt a safety mode if thecontrol unit determines that the first control signals do not meet atleast the first reliability criteria.
 2. The device of claim 1,characterized in that the first body part is the passenger's head,wherein, a) the control unit determines basic data on the basis ofgeometric dimensions and/or size ratios and/or features of the faceand/or gestures detected by at least the second sensor; and b) the firstreliability criteria are determined using the basic data.
 3. The deviceaccording to claim 1, characterized in that the control unit is furtherdesigned to determine the first control signals using at least thesecond sensor signals; and e) to adopt the safety mode when the controlunit determines that the first control signals do not meet at least thefirst and/or second reliability criteria, wherein the first and/orsecond reliability criteria are determined by accuracy and/orreliability analysis of at least the first sensor signals and the secondsensor signals.
 4. The device according to claim 1, characterized inthat the control unit determines the fulfillment of the firstreliability criteria by implementing machine learning, comprisingclassification, or by implementing a neural network.
 5. The deviceaccording to claim 1, characterized in that the second and/or a thirdsensor comprising a vitality data sensor, is arranged on a body part orbetween two body parts and detects vitality parameters as characterizingfeatures and outputs second and/or third sensor signals, wherein thecontrol unit receives the second and/or third sensor signals and, if avitality parameter value range is not met by the second and/or thirdsensor signals, the second reliability criteria are not met.
 6. Thedevice according to claim 1, characterized in that at least one firstsensor assembly comprising an ultrasonic sensor assembly and/or a LIDARsensor assembly and/or an image-based sensor assembly and/or a RADARsensor assembly, is mounted on the vehicle for sensing the environmentand is designed to output first sensor assembly signals, wherein thecontrol unit receives the first sensor assembly signals and determinesthe first control signals using the first sensor assembly signals. 7.The device according to claim 1, characterized in that a remotecommunication device is arranged on the vehicle and is designed toperform signal exchange with the control unit and mobile providerdevices for performing video communication and/or remote healthcondition monitoring of the passenger.
 8. The device according to claim1, characterized in that the first and/or a fourth sensor is mounted onor in a wearable and is designed to detect the first and/or a secondbody part or their features of the passenger of the vehicle and tooutput first or fourth sensor signals, wherein, by using said first orfourth sensor signals, the first control signals for controlling thevehicle are determined.
 9. The device according to claim 1,characterized in that the first and/or a fifth sensor comprising a braincontrol unit interface or input device is arranged on and/or in the headof the passenger and is designed to output first or fifth sensorsignals, wherein, by using said first or fifth sensor signals, the firstcontrol signals for controlling the vehicle are determined.
 10. Thedevice according to claim 1, characterized in that the first and/or asixth sensor comprising a speech input sensor is designed to outputfirst and/or sixth sensor signals, wherein the control unit receives thefirst and/or sixth sensor signals and, by using said first and/or sixthsensor signals, determines the first control signals for controlling thevehicle.
 11. The device according to claim 10, characterized in that thecontrol unit is designed to use an implementation of neural networks,when processing the first and/or sixth sensor signals for determiningthe first control signals for controlling the vehicle.
 12. The deviceaccording to claim 1, characterized in that the wearable and/or asmartphone is designed and arranged to output seventh sensor signalscomprising passenger destination input sensor signals, wherein thecontrol unit transmits available destination input selection data to thewearable and/or the smartphone for passenger destination input as wellas receives the seventh sensor signals and, by using said seventh senorsignals, determines the first control signals.
 13. The device accordingto claim 1, characterized in that an eighth sensor comprising a positionsensor or a sensor for locating inside buildings, is arranged anddesigned to output eighth sensor signals, wherein the control unitreceives the eighth sensor signals and, by using said eighth sensorsignals, determines the first control signals.
 14. The device accordingto claim 1, characterized in that vehicle dynamics sensors, are arrangedon the vehicle are designed to output ninth sensor signals, wherein thecontrol unit receives the ninth sensor signals and, by using said ninthsensor signals, determines the first control signals.
 15. A vehicle,comprising: a device including; at least one first sensor including aninertial navigation which is designed and arranged to detect at leastone first body part, including an absolute position and/or positionand/or rotation and/or translation, of a passenger of the vehicle and tooutput first sensor signals; at least one second sensor including animage-based sensor which is designed and arranged to detect at least thefirst body part of the passenger and/or their features, including theirabsolute position and/or position and/or rotation and/or translation,and to output second sensor signals; and a control unit designed to; (a)receive the first and second sensor signals; (b) determine first controlsignals for controlling the vehicle based at least on the first sensorsignals; (c) determine whether the first control signals meet at leastfirst reliability criteria based at least on the second sensor signals;and (d) adopt a safety mode if the control unit determines that thefirst control signals do not meet at least the first reliabilitycriteria; and one or more vehicle actuators including controllablemotors, arranged to drive the vehicle and designed to receive andprocess the control signals of the device.
 16. A method for controllinga device for navigating and/or guiding the path and/or stabilizing avehicle comprising: a) receiving at least first sensor signalscomprising inertial navigation-based sensor signals, which describe atleast a first body part, including its absolute position and/or positionand/or rotation and/or translation, of a passenger of the vehicle; andb) receiving second sensor signals comprising an image-based frontsensor, which is designed and arranged to describe at least the firstbody part or its characterizing features, including their absoluteposition and/or position and/or rotation and/or translation, of thepassenger of the vehicle; c) determining first control signals forcontrolling the vehicle based at least on the first sensor signals; d)determining, via a control unit, whether the first control signals meetat least first reliability criteria based on at least the second sensorsignals; and e) adopting a safety mode when the control unit determinesthat the control signals do not meet at least the first reliabilitycriteria.
 17. The method according to claim 16, characterized in that,based on evaluation of the second and/or third sensor signals comprisingvitality parameters of the passenger, the second reliability criterionis not met if a vitality parameter value range is violated.
 18. Themethod according to claim 16, characterized in that first controlsignals for controlling the vehicle are determined based on evaluationof the first and/or fourth sensor signals, comprising an eye featuredetection unit or inertial navigation data-based sensor signals.
 19. Themethod according to claim 16, characterized in that the first controlsignals for controlling the vehicle are determined based on evaluationof first sensor assembly signals of at least one environment-sensingsensor assembly and/or eighth sensor signals of a position sensor usingpassenger destination input sensor signals.
 20. The method of claim 19,wherein the at least one environment-sensing sensor assembly includesone or more of an ultrasonic sensor assembly, a LIDAR sensor assembly,an image-based sensor assembly or a RADAR sensor assembly.